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TIi is  is  the  final  report  of  work  performed  on  two  projects, 
entitled  "Development  and  Evaluation  of  Self-Instructional 
Materials  on  the  Construction  of  Skill  Qualification  Tests"  and 
"Development  and  Evaluation  of  Self-Instructional  Materials  on 
the  Validation  of  Skill  Qualification  Tests."  The  report  de- 
scribes the  development  of  the  SQT  Development  Workshop.  The 
workshop  materials  are  distributed  within  the  Army  by  TRADOC. 

The  work  was  conducted  by  the  Human  Resources  Research 
Organization  (HumRRO)  under  Contract  No.  DAHC  19-77-C-0005  and 
Contract  No.  DAHC  19-77-C-0013  with  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences.  The  Radcliff, 
Kentucky  Office  of  HumRRO 's  Western  Division  performed  the  work. 
Dr.  Howard  McFann  is  Vice-President  and  Director  of  the  HumRRO 
Western  Division.  Mr.  William  Osborn,  in  his  role  as  Director 
of  the  Radcliff  Office  and  Project  Director,  contributed  at  all 
stages  of  the  projects. 

Dr.  Milton  H.  Maier,  the  Contracting  Officer’s  Technical 
Representative,  provided  administrative  guidance  throughout  the 
projec  ts . 

These  projects  could  not  have  been  successful  without  the 
full  cooperation  of  the  Individual  Training  and  Evaluation  Direc- 
torate (ITED),  Training  Developments  Institute  (TDI)  and  staff  and 
faculty  development  personnel  at  the  various  schools. 
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SUMMARY 


As  part  of  the  Army's  restructuring  of  the  Enlisted  Personnel 

Management  System,  the  use  of  Skill  Qualification  Tests  (SQT)  has 

been  introduced  as  the  means  of  determining  individual  soldier  pro- 

* 

ficiency.  Implementation  of  SQT  constitutes  a significant  change 
in  development  and  application  procedures  from  previous  methods. 
Development  of  SQT  is  the  responsibility  of  Test  Development 
Agencies  (TDA)  that  are  proponent  for  MOS.  SQT  developers  at  TDA 
are  subject  matter  experts  for  MOS,  but  often  are  not  familiar  with 
the  procedures  required  for  SQT  development.  Cuidance  for  SQT 
developers  has  been  disseminated  in  the  Guidelines  for  Development 
of  Skill  Qualification  Tests  (Individual  Training  and  Evaluation 
Directorate,  U.S.  Army  Training  Support  Center,  December  77). 

Early  experience  with  SQT,  however,  indicated  the  need  for  train- 
ing on  the  principles  presented  in  the  Guidelines.  The  purpose  of 
this  project  was  to  convert  the  Guidelines  to  a self-contained 
self-instructional  workshop  for  SQT  developers,  to  tryout  the 
workshop  at  selected  TDA,  and  to  revise  workshop  materials  based 
on  the  tryouts. 

The  workshop  had  to  accomodate  three  constraints:  it  had  to 
be  exportable  to  the  extent  that  staff  and  faculty  development  per- 
sons at  the  TDA  could  be  trained  as  course  managers  and  conduct  the 
workshop  as  needed  to  meet  TDA  requirements;  self-paced,  to  allow 
Individuals  with  varying  amounts  of  experience  with  SQT  to  concen- 
trate on  unfamiliar  areas;  and  limited  to  ten  days,  to  fit  the  ticht 
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scheduling  within  TDA.  It  was  also  decided  that  the  workshop  would 
be  modelled  on  the  principles  of  criterion-referenced  instruction. 

The  Guidelines  contains  procedures  on  nine  tasks  required  in 
the  construction  and  validation  of  SQT.  Analysis  of  the  nine  tasks 
resulted  in  the  delineation  of  34  skills  which  are  subordinate  to 
the  nine  tasks.  For  each  of  the  34  subordinate  skills,  a module 
was  prepared,  containing  explanation,  examples,  and  activities 
addressing  that  skill.  Criterion  tests  and  accompanying  evaluation 
sheets  were  also  prepared  for  each  skill,  through  which  workshop 
participants  demonstrated  mastery  of  the  skill  according  to  pre- 
determined standards. 

The  workshop  materials  were  reviewed  by  staff  of  the  Individual 
Training  and  Evaluation  Directorate  (ITED),  which  is  responsible 
for  review  of  all  SQT  products,  and  by  the  Training  Developments 
Institute  (TDI) , which  is  responsible  for  staff  and  faculty  develop- 
ment and  training  at  the  TDA.  Based  on  their  review,  and  on  comments 
from  the  first  tryout  of  the  workshop,  the  materials  were  revised. 

The  workshop  was  then  conducted  for  20  representatives  of  13  TDA 
selected  by  TDI.  After  attending  a course  manager  seminar,  these 
persons  acted  as  course  managers,  under  TDI,  ITED,  and  project  staff 
supervision,  at  successive  implementations  of  the  workshop  at  13  TDA. 
At  these  workshops,  213  participants  from  22  TDA  were  trained. 

As  participants  completed  the  workshop,  they  were  asked  to 
respond  to  an  end-of-course  critique,  eliciting  their  reactions  to 
the  format,  content,  and  impact  of  the  workshop.  At  all  workshops, 
responses  were  overwhelmingly  positive.  Participants'  additional 


comments  were  used  In  making  final  revisions  to  the  workshop 
materials,  the  workshop  was  completed  in  an  average  of  7.76  work- 
ing days,  with  98.72  of  participants  completing  within  the  10  days 
allotted. 

the  13  iterations  of  the  workshop  resulted  in  the  certifica- 
tion of  18  course  managers  at  14  IDA.  Each  of  these  tDA  is  now 
responsible  for  implementing  the  workshop  as  needed  for  SQT  develop- 
ers. Camera-ready  copies  of  all  workshop  materials  were  provided 
to  certified  course  managers,  who  reproduce  and  distribute  the 
materials  for  subsequent  workshops. 
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CHAPTER  1 
INTRODUCTION 


BACKGROUND 

As  part  of  the  Army's  restructuring  of  the  Enlisted  Personnel 
Management  System  (EPMS) , a new  method  of  determining  individual 
soldier  proficiency  through  testing  has  been  introduced.  This 
method  centers  on  the  use  of  Skill  Qualification  Tests  (SQT)  as  the 
means  of  testing.  Implementation  of  SQT  constitutes  a significant 
change  in  development  and  application  procedures  from  previous 
methods . 

Development  of  an  SQT  is  the  responsibility  of  the  school  or 
other  agency  that  is  the  proponent  for  the  MQS.  There  are  34  such 
Test  Development  Agencies  (TDA) . Review  and  production  of  SQT  pro- 
ducts at  the  Training  and  Doctrine  Command  (TRADOC)  level  is  the 
responsibility  of  the  Individual  Training  and  Evaluation  Directorate 
(ITED).  This  division  of  labor  helps  assure  that  test  developers 
will  have  the  required  level  of  subject  matter  expertise,  but  leaves 
open  the  possibility  that  they  will  lack  the  test  development  skills 
required  by  the  SQT  program. 

Early  implementation  of  SQT  was  both  large-scale  and  rapid. 

The  need  for  guidance  for  test  developers  was  initially  answered  by 
the  Handbook  for  Development  of  Skill  Qualification  Tests, 1 which 
was  later  published  as  Guidelines  for  Development  of  Skill  Qualifi- 
cation Tests.2  This  document  presented  procedures  to  perform  tasks 
which  ITED  had  identified  as  sources  of  recurring  problems  in  SQT 
development . 

In  addition  to  the  Guidelines,  however,  a need  was  perceived 
for  a controlled,  systematic  approach  to  training  developers  to 
apply  the  principles  in  the  Guidelines.  The  SQT  Development  Work- 
shop was  proposed  as  a means  to  provide  monitored  practice  in  the 
skills  required  to  construct  and  validate  an  SQT. 


!William  C.  Osborn,  Roy  C.  Campbell  and  J.  Patrick  Ford.  Handbook 
for  Development  of  Skill  Qualification  Tests,  HumRRO  Final  Report 
77-1,  January  1977. 

^Individual  Training  and  Evaluation  Directorate,  U.S.  Army  Training 
Support  Center.  Guidelines  for  Development  of  Skill  Qualification 
Tests.  December  1977. 


The  original  plan  called  for  separate  workshops  to  cover  the 
construction  and  validation  of  SQT.  Although  construction  and 
validation  at  first  appeared  to  be  distinct  processes,  closer 
analysis  revealed  that  they  are  inseparable  in  criterion-referenced 
testing.  This  is  especially  true  of  the  SQT  system  where  validation 
is  a tool  for  revising  and  refining  a test.  Therefore,  the  work 
focussed  on  preparing  one  workshop. 


OBJECTIVE 

The  objective  of  the  project  was  twofold: 

. To  convert  Guidelines  for  Development  of  SQT 
to  a self-contained  self-instruction  workshop. 

. To  try  out  and  revise  workshop  materials  at 
selected  TM. 

The  workshop  had  to  accommodate  three  constraints.  The  first 
of  these  is  that  it  had  to  be  exportable.  While  responsibility  for 
development  of  the  workshop  was  assumed  by  TRADOC,  the  ultimate  respon- 
sibility for  implementing  the  workshop  rested  with  each  TDA.  Since 
TDA  traditionally  experience  considerable  turnover  among  SQT  personnel, 
they  must  be  able  to  repeat  the  workshop  as  often  as  their  needs  dic- 
tate. 


The  decision  was  made  to  make  the  workshop  a part  of  the  total 
Faculty  Development  Program  under  the  direction  of  the  Training 
Developments  Institute  (TD1) . At  the  TDA  or  school  level,  the  workshop 
would  be  the  responsibility  of  the  staff  and  faculty  development 
unit.  The  requirement  then,  was  that  the  workshop  be  exportable  to 
the  extent  that  it  could  be  taught  to  staff  and  faculty  development 
personnel,  who  would  then  act  as  course  managers  at  their  TDA  and 
conduct  the  workshop  as  needed  to  meet  their  own  requirements. 

The  second  constraint  was  that  the  workshop  be  self-paced.  While 
the  TRADOC  training  philosophy  incorporates  self-pacing  in  its 
instructional  model,  this  was  not  the  only  basis  for  this  requirement. 
People  assigned  to  SQT  development  can  be  expected  to  have  a variety 
of  experience.  Most  will  know  very  little  about  testing.  Other 
people  will  know  little  about  the  practical  limitations  of  SQT,  but 
be  skilled  test  developers.  Thus,  as  the  need  to  learn  about  various 
aspects  of  criterion  test  development  in  general  and  SQT  in  particu- 
lar will  vary,  the  workshop  had  to  allow  individuals  to  work  at 
their  own  pace. 

The  third  constraint  related  to  the  time  of  the  workshop.  TDI 
recommended  that  ten  days  be  established  as  the  maximum  length  of 
the  workshop.  The  limit  was  needed  because  of  anticipated  resis- 
tance of  managers  to  allow  people  to  be  away  from  their  desks  for 
more  than  ten  days  to  learn  to  develop  an  SQT. 
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Development  and  implementation  of  an  exportable,  self-paced, 
ten-day  workshop  proceeded  through  three  phases: 

. Development  of  initial  workshop  materials. 

. Implementation  of  workshop  at  selected  TDA. 

. Evaluation  of  workshop. 
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CHAPTER  2 

DEVELOPMENT  OF  INITIAL  WORKSHOP  MATERIALS 


As  part  of  the  total  Faculty  Development  Program,  TDI  had 
successfully  Implemented  the  Criterion-Referenced  Instruction  (CRI) 
Workshop,  developed  by  Mager  Associates.  The  CRI  workshop  has 
become  the  foundation  for  a family  of  staff  and  faculty  develop- 
ment programs  that  are  intended  to  provide  the  necessary  in-house 
training  capability  in  each  TRADOC  training  facility. 

According  to  the  CRI  model  the  overall  objective  for  a training 
program  is  broken  into  subordinate  objectives,  and  training  is  pre- 
sented in  modules  corresponding  to  these  subordinate  objectives. 
Within  some  limits,  participants  choose  the  sequence  in  which  they 
will  work  through  the  modules.  At  the  beginning  of  each  module, 
the  objective  for  that  module  is  stated,  the  criterion  test  is 
described,  and  resource  references  for  the  material  are  listed. 

Each  participant  decides  how  much  study  and  practice  he  needs  to 
pass  the  criterion  test.  A course  manager  monitors  student  progress, 
evaluates  criterion  tests,  and  serves  as  a learning  resource  when 
required  by  the  student. 

Workshop  materials  were  developed  to  conform  to  both  the  for- 
mat and  the  content  of  this  model.  Two  tasks  comprised  development 
of  the  workshop  materials: 

. Prepare  materials. 

. Review  and  tryout  materials. 


PREPARE  MATERIALS 

The  scope  of  the  workshop  was  dictated  by  the  scope  of  Guide- 
lines for  Development  of  SQT.  The  purpose  of  the  Guidelines  is  to 
set  forth  "procedures  to  be  followed  by  test  development  agencies 
in  planning,  constructing,  and  field  testing  SQT.”1  In  the  Guide- 
lines the  three  aspects  of  the  purpose  are  broken  into  nine  major 
tasks,  listed  in  the  first  column  of  Table  1.  The  nine  tasks  were 
derived  rationally  through  analysis  of  procedures  required  for  SQT 
development.  Necessarily,  then,  the  Guidelines  and  the  workshop 
are  prescriptive  rather  than  descriptive;  that  is,  they  are  based 
on  projections  of  what  should  be  done,  rather  than  on  an  empirical 
study  of  how  successful  SQT  development  was  proceeding  at  various 
TDA. 


1 Ibid. , p.  1-1. 
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Table  1 


SUBORDINATE  OBJECTIVES  DERIVED  FROM 
GUIDELINES  FOR  DEVELOPMENT  OF  SQT 


Guidelines  Tasks 
Select  Tasks  for  Testing 

Review  Task  Analysis 
Allocate  Tasks  to  Components 
Construct  Hands-On  Component 


Tryout  Hands-On  Component 


Construct  Written  Component 


Validate  Written  Component 


Construct  Performance 
Certification  Component 

Prepare  SQT  Notice 


Subordinate  Objectives 

Identify  Sources  of  Information 
Select  Tasks  for  Testing 

Review  Task  Analysis 

Allocate  Tasks  to  Components 

Determine  Product  or  Process 
Construct  Process  Performance 
Measures 

Construct  Product  Performance 
Measures 

Determine  Tolerance  and  Time 
Construct  HOC  Performance  Measures 
Determine  Test  Conditions 
Prepare  Examinee  Instructions 
Prepare  Scorer  Instructions 

Locate  Defects  (Expert  Tryout) 

Compute  Scoring  Agreement 
Determine  Feasibility 
Prepare  Station  Load  Table 
Revise  Hands-On  Scorable  Unit 

Determine  Written-Performance  or 
Performance-Based 
Prepare  Written-Performance  Items 
Prepare  Performance-Based  Items 
Set  Scoring  Standards 
Construct  Written  Scorable  Unit 

Collect  Self-Ratings 
Locate  Defects  (Expert  Tryout) 
Validate  Written  SU  Against  Hands-On 
Select  Validation  Option 
Select  Sample  for  WC  Validation 
with  Self-Ratings 

Determine  Acceptability  of  Written  SU 
Validate  Written  SU:  Self-Ratings 
Validate  Written  SU:  Supervisor 
Ratings 

Validate  Written  SU:  Panel  of 
Experts 

Revise  Written  Items 

Construct  Performance  Certification 
Component 

Prepare  SQT  Notice 


The  first  step  in  preparing  the  workshop  materials  was  to  | 

identify  the  subordinate  objectives  within  the  nine  tasks  addressed  \ 

by  the  Guidelines.  The  34  subordinate  objectives  that  emerged  are  ! 

listed  in  the  second  column  of  Table  1.  Generally,  if  the  Guide- 
lines task  required  several  distinct  skills,  each  skill  was  con- 
sidered to  be  a subordinate  objective.  If  the  Guidelines  task  was 
not  complex,  the  task  by  itself  was  considered  to  be  a subordinate 
objective.  Some  skills  were  subordinate  to  more  than  one  Guide-  < 

lines  task  (e.g.,  constructing  performance  measures  was  a skill  j 

required  for  construction  of  hands-on  and  performance  certification  ] 

components)  . In  such  cases  the  skill  was  considered  to  be  sub-  ^ 

ordinate  only  to  the  task  with  which  it  was  discussed  in  the 

Guidelines.  | 

■i 

The  second  step  was  to  prepare  a module  for  each  subordinate 
objective.  A module  consists  of  text  that  amplifies  the  decisions  3 

and  actions  in  the  Guidelines,  examples  that  show  how  the  decisions 
and  actions  relate  to  specific  situations,  and  activities  that  have 
the  participant  apply  the  decisions  and  actions  to  a situation. 

The  examples  and  activities  in  the  modules  make  use  of  sample  tasks,  t j 

which  were  selected  with  two  criteria  in  mind:  I j 

! ] 

. The  task  must  illustrate  the  princlple(s) 
being  discussed.  1 

. The  task  must  be  familiar  to  participants.  i 

I 

For  the  most  part  sample  tasks  are  common  military  tasks  (e.g.,  ] 

Fire  a Claymore  Mine,  Prepare  a Wrist  Cast,  Complete  DA  Form  2404).  j 

j 

The  activities  (or  practical  exercises)  serve  two  purposes. 

First,  they  allow  participants  to  practice  the  skills  being  covered. 

Second,  the  activities  illustrate  some  of  the  variations  possible  ‘ 

when  the  skill  is  applied  to  real  world  situations.  Thus,  they 
serve  not  only  a practical  function  but  also  as  a learning  source. 

Each  activity  provides  space  for  participants  to  analyze  the  problem 
and  record  their  own  answers,  and  is  followed  by  feedback  which 
includes  rationale  for  the  provided  answer. 

The  third  step  in  developing  the  materials  was  to  prepare  a 
criterion  test  for  each  module.  In  some  cases  the  criterion  tests 
are  standard,  that  is,  all  participants  work  with  the  same  situation 
or  task.  In  other  cases  participants  work  with  tasks  or  material 
of  their  own  choosing  from  their  selected  MOS.  The  individualized 
approach  was  chosen  as  much  as  possible  within  the  time  constraints 
for  two  reasons : 
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. It  would  demonstrate  the  adaptability  of  the 
procedures • 

. The  obvious  job  relevance  of  the  materials 
would  help  maintain  participants'  interest. 

About  one-half  of  the  criterion  tests  require  participants  to 
work  with  their  own  MOS-relevant  tasks.  However,  these  tests  cover 
all  phases  of  constructing  a written  and  hands-on  component  of  an 
SQT.  In  fact,  each  participant  must  produce  two  complete  hands-on 
SU  and  two  written  SU  to  complete  the  workshop. 

Standard  criterion  tests  are  used  for  two  kinds  of  situations. 
The  first  situation  requires  participants  to  make  preliminary 
decisions  on  standard  tasks  before  they  make  similar  decisions  on 
their  selected  tasks.  In  Hands-On  and  Written  Component  Construc- 
tion, participants  work  with  standard  criterion  tests  on  the  first 
four  modules,  and  then  with  their  own  tasks  in  capper  modules.  The 
second  situation  for  standard  criterion  tests  pertains  to  tests  of 
validation  and  tryout  skills.  The  rationale  for  this  application 
of  the  standard  criterion  test  approach  is  discussed  later  under 
Review  and  Tryout  of  Materials. 

The  fourth  step  in  developing  the  materials  was  to  prepare  an 
evaluation  sheet  for  each  criterion  test.  If  the  criterion  test 
was  standard,  the  appropriate  solution  or  solutions  were  listed. 

If  the  criterion  test  was  based  on  tasks  participants  chose,  the 
evaluation  sheet  was  designed  as  a checklist  a reviewer  would  follow 
to  check  the  work.  One  of  three  people  was  designated  to  check  the 
work  according  to  the  following  rules: 

. If  the  test  is  standard,  the  participant  is 
authorized  to  check  his  own  work  (self 
sign-off) . 

. If  the  test  is  based  on  tasks  participants 
choose  but  requires  no  judgment  to  review, 
a colleague  is  authorized  to  check  the  work. 

. If  the  test  is  based  on  tasks  participants 
choose  and  requires  judgment  by  the  reviewer, 
the  course  manager  must  check  the  work. 

The  rules  for  identifying  the  reviewer  of  the  criterion  test 
were  modified  after  experience  with  the  tryouts.  Some  participants 
concluded  that  modules  they  could  review  by  themselves  were  less 
important  than  other  modules,  so  they  skipped  them  or  worked  only 
superficially.  In  most  cases  their  error  was  corrected  by  later 


criterion  teste  that  required  them  to  apply  the  skills  from  the 
previous  module.  For  the  final  draft  of  the  materials  the  rule  for 
designating  a criterion  test  as  a self  sign-off  was: 


i 


1 


[< 


if 


. If  the  test  is  standard  and  the  skills  or 
knowledge  are  required  by  a later  module  that 
the  course  manager  checks,  the  participant 
is  authorised  to  check  his  own  work.  Other- 
wise, the  course  manager  will  review  standard 
tests . 

Similarly,  colleague  sign-offs  were  further  limited  to  individualized 
tests  that  cover  skills  and  knowledges  that  the  course  manager  checks 
in  a later  module. 

Finally,  supplementary  materials  were  prepared  for  course  con- 
trol and  participant  orientation.  These  materials  include: 

. The  Course  Map,  which  shows  the  necessary 
sequence  in  which  certain  modules  may  be 
studied. 

. The  Personal  Progress  Summary,  on  which 
completed  criterion  tests  are  signed  off. 

. The  Overview  of  SQT  Development  Workshop, 
which  explains  the  workshop  procedures  and 
lists  acronyms  and  abbreviations  used  in 
the  workshop. 

. An  additional  module,  "Review  SQT  Develop- 
ment Process,"  which  introduces  the  partici- 
pant to  the  Guidelines.  The  criterion  test 
for  this  module  requires  participants  to 
identify  their  role  in  SQT  development,  and 
to  select  one  MOS  with  which  to  work  through- 
out the  workshop.  This  criterion  test  has  a 
covert  objective  as  well:  to  give  course 
managers  a feel  for  the  various  MOS  with  which 
participants  will  be  working,  and  for  the 
way  in  which  the  TDA  has  organized  SQT 
developers . 

The  workshop  materials  are  being  distributed  within  the  Army  by 
TRADOC;  copies  are  available  at  AR1, 
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REVIEV.  AND  TRYOUT  MATERIALS 


As  modules  were  completed,  they  were  submitted  for  review  to 
representatives  of  TDI,  ITED  and  the  COTR.  Although  there  was  some 
overlap,  TDI  reviewers  checked  the  practicability  of  the  activities 
and  criterion  tests;  ITED  reviewers  checked  consonance  witli  policy; 
and  the  COTR  checked  the  scope  and  coverage  of  the  material.  Mater- 
ials were  then  revised  to  incorporate  the  reviewers'  recommendations. 

The  first  tryout  of  the  materials  was  conducted  at  Ft.  Eustis, 
Virginia  with  two  representatives  of  TDI,  three  test  psychologists 
from  ITED,  seven  SQT  developers  from  the  Transportation  School,  and 
one  representative  of  the  COTR.  Three  members  of  the  project  staff 
managed  the  workshop. 

Materials  were  again  revised  based  on  experience  with  the  tryout. 
The  major  revision  was  to  eliminate  required  group  work  from  the 
activities  and  criterion  tests.  During  development  of  the  materials 
on  validating  SQT,  the  intent  had  been  to  require  participants  to 
conduct  a validation  study  using  other  participants  as  the  sample 
for  the  tiands-on  and  written  SU  they  had  developed  earlier.  This  was 
found  to  be  Impractical  for  four  reasons: 

. Commanders  were  reluctant  to  release  equip- 
ment for  the  hands-on  validation  tryouts. 

. Participants  resented  interrupting  their 
work  to  serve  as  subjects  for  a colleague's 
validation  study. 

. The  studies  added  at  least  one  day  to  each 
participant's  completion  time. 

. The  diversity  of  the  participants  made  it 
virtually  Impossible  to  obtain  a pool  of 
people  who  could  perform  MOS  specific 
tasks . 

For  these  reasons  it  was  concluded  that  the  benefits  did  not  justify 
the  expense  of  actual  tryouts.  In  the  final  materials,  validation 
criterion  tests  present  data  from  tryouts  and  require  participants 
to  analyze  the  data  and  prescribe  revisions  to  be  made. 


CHAPTER  3 


IMPLEMENTATION  OF  WORKSHOP  AT  SELECTED  TDA 


The  workshop  was  implemented  at  the  TDA  through  performance  of 
four  tasks: 

. Train  course  managers. 

. Conduct  workshop  at  selected  TDA. 

. Evaluate  management  aspects. 

. Revise  course  materials. 


TRAIN  COURSE  MANAGERS 

As  discussed  earlier,  the  workshop  was  intended  to  be  managed 
ultimately  by  staff  and  faculty  development  personnel  at  the  various 
TDA.  To  meet  the  TDI  requirements  as  a certified  course  manager, 
each  course  manager  trainee  first  had  to  complete  the  workshop  man- 
aged by  the  project  staff,  and  then  serve  as  a course  manager  under 
the  supervision  of  staff  and  TDI  personnel. 

The  first  step  was  to  conduct  the  workshop  for  20  representa- 
tives of  13  TDA  selected  by  TDI.  The  workshop  was  conducted  at  the 
Xerox  International  Center  for  Training  and  Management  Development, 
near  Leesburg,  Virginia.  The  workshop  was  managed  by  two  people 
from  TDI  and  one  ITED  representative,  all  of  whom  had  completed  the 
workshop  during  the  initial  tryout,  as  well  as  three  members  of  the 
project  staff. 

As  participants  finished  the  workshop,  they  attended  a course 
manager  seminar.  At  that  time  they  were  instructed  in  the  steps 
needed  to  set  up  the  workshop  at  their  TDA.  A member  of  the  project 
staff  then  led  a discussion  of  specific  modules  and  criterion  tests. 


CONDUCT  WORKSHOP  AT  SELECTED  TDA 

The  first  workshop  at  each  TDA  was  conducted  by  a team  of 
course  managers.  The  team  consisted  of  at  lea9t  one  project  staff 
member,  at  least  one  TDI  representative,  at  least  one  staff  and 
faculty  development  member  from  the  host  TDA  who  had  attended  the 
Leesburg  training,  and  an  ITED  test  psychologist  who  served  as  an 
available  resource  to  participants  and  provided  guidance  on  policy 
questions. 
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Thirteen  workshops  were  conducted  at  the  locations  listed  in 
Table  2.  A total  of  213  participants  from  22  TDA  completed  the 
workshops  conducted  at  the  various  TDA.  Most  participants  (76%) 
were  directly  involved  in  SQT  development.  Their  previous  experi- 
ence with  SQT  ranged  from  one  month  to  over  three  years.  The  median 
among  experienced  developers  was  5.5  months.  The  24%  with  no 
previous  experience  included  people  newly  assigned  to  SQT,  staff 
and  faculty  development  personnel,  and  representatives  of  the 
Directorate  of  Evaluation. 

As  a result  of  the  implementation  workshops,  14  TDA  have  the 
capability  to  conduct  the  workshop  without  outside  assistance.  This 
means  that  at  least  one  member  of  their  staff  and  faculty  develop- 
ment unit  has  completed  the  workshop  and  has  helped  manage  the 
workshop  while  being  observed  by  the  TDI  SQT  Team.  Each  TDA  with 
that  capability  is  marked  with  an  asterisk  in  Table  2. 


EVALUATE  MANAGEMENT  ASPECTS 

During  the  conduct  of  the  workshop  at  the  various  TDA,  project 
staff  personnel  observed  a number  of  different  management  aspects 
of  the  workshop.  While  some  of  these  aspects  were  controlled  and 
some  varied  as  a result  of  circumstances,  they  all  provided  some 
insight  into  efficient  workshop  operations.  The  major  aspects  of 
management  are: 

. Participant  to  course  manager  ratio. 

. Participant  attendance. 

. Control  of  tests  and  evaluation  sheets. 

Participant  to  Course  Manager  Ratio.  The  ratio  varied  among 
workshops  from  2:1  to  slightly  over  6:1.  The  optimum  ratio  appears 
to  be  around  5:1.  This  results  in  minimal  participant  waiting  time 
for  module  sign-offs  and  maximum  utilization  of  course  manager 
personnel  without  encountering  course  manager  fatigue.  This  ratio 
is  advisable  for  at  least  the  first  three  days  of  the  workshop  when 
the  course  manager  load  is  the  heaviest.  After  this,  participant 
workload  tends  to  spread  out  and  a ratio  of  8:1  is  not  uncomfortable. 

Participant  Attendance.  Although  participants  were  adviced 
that  their  progress  through  the  workshop  was  self-paced,  all  started 
at  the  same  time  and  were  expected  to  complete  the  workshop  within 
the  ten-day  time  period.  Participants  who  attempted  to  work  at 
their  regular  job  while  completing  the  workshop,  or  who  attempted 
to  work  at  their  normal  place  of  duty  instead  of  the  workshop  area, 
tended  to  fall  behind.  Some  were  subsequently  dropped  from  the 
workshop . 


Table  2 


I 

j LOCATIONS  AND  PARTICIPANTS  OF  SQT  DEVELOPMENT  WORKSHOP 

I 


Workshop  Location 

TDA  Attending 

Participants 

Trained 

Ft.  Eustis,  VA 

Transportation  School* 

12 

School  of  Music 

1 

Ft.  Bliss,  TX 

Air  Defense  Artillery  School* 

17 

Ft.  Gordon,  GA 

Signal  School* 

18 

Ft.  Huachuca,  AZ 

Army  Intelligence  School* 

10 

Redstone  Arsenal,  AL 

Missile  and  Munition  School* 

12 

Military  Police  School 

3 

Criminal  Investigation  Division 
Comnand 

2 

Ft.  Benning,  GA 

Infantry  School* 

7 

Aviation  School 

1 

Ft.  Lee,  VA 

Quartermaster  School* 

17 

Aberdeen  Proving 

Grounds,  MD 

Ordnance  and  Chemical  School* 

11 

Ft.  Benjamin 

Harrison,  IN 

Institute  of  Administration* 

15 

Defense  Information  Service 

2 

Army  Intelligence  School  (ASA) 

2 

Ft.  Sill,  OK 

Field  Artillery  School* 

11 

Ft.  Belvoir,  VA 

Engineer  School* 

18 

Chaplain  School 

2 

Defense  Mapping  School 

1 

Ft.  Knox,  KY 

Armor  School* 

27 

Institute  for  Military  Assistance*  3 

Ft.  Sam  Houston,  TX 

Academy  of  Health  Sciences* 

21 

*TDA  certified  to  conduct  workshop. 
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Few  participants  are  willing  or  able  to  refuse  their  super- 
visors when  daily  work  interferes  with  the  workshop.  Maximum 
workshop  benefit  is  obtained  when  participants  are  removed, 
geographically  and  psychologically,  from  their  normal  work  areas. 
On  the  other  hand,  the  workshop  schedule  should  not  be  so  rigid 
that  participants  are  unable  to  schedule  their  own  time.  The 
recommended  approach  is  to  schedule  the  workshop  for  six  hours 
per  day  and  require  participants  to  be  at  the  site  during  that 
period.  While  the  workshop  can  be  conducted  on  an  open-entry, 
open-exit  basis,  experience  during  the  implementation  phase  sug- 
gests that  such  a delivery  mode  will  increase  average  time  and 
decrease  the  percentage  of  participants  who  complete  the  work- 
shop . 

Control  of  Tests  and  Evaluation  Sheets.  Two  major  methods 
were  used  to  control  criterion  tests  and  evaluation  sheets.  In 
one,  the  tests  and  evaluation  sheets  were  placed  in  separate 
boxes  in  the  room  and  participants  were  free  to  get  either 
whenever  they  wished.  In  the  other  method,  participants  were 
given  all  material,  including  tests  and  evaluation  sheets,  at 
the  start  of  the  workshop.  In  both  cases,  participants  could 
copy  responses  from  the  evaluation  sheets  if  the  wanted  to. 

Most  participants  preferred  control  of  the  sheets  located  in 
separate  boxes.  Some  participants  who  had  all  materials  com- 
plained that  the  temptation  to  look  at  the  evaluation  sheet  for 
answers,  instead  of  seeking  assistance,  was  almost  overwhelming. 
It  should  be  noted  that,  based  on  the  evaluation  of  the  target 
audience  for  this  workshop,  no  trial  was  made  of  stricter  control 
of  the  tests  and  evaluation  sheets.  Yet  on  some  occasions,  par- 
t icipants  indicated  a desire  for  more  rigid  control  of  these 
documents.  Usually  this  was  prompted  by  a belief  that  others 
were  not  using  the  evaluation  fairly,  thereby  somehow  penalizing 
the  respondent. 

i. 


REVISE  COURSE  MATERIALS 

Concurrent  with  the  conduct  of  the  workshops,  project  staff 
monitored  student  reaction  both  formally,  through  an  end-of-course 
critique,  and  informally,  through  observation  and  discussion.  As 
a result,  several  revisions  were  made  affecting  the  content  and 
organization  of  course  materials. 

. The  amount  of  arithmetical  work  in  the  vali- 
dation modules  was  scaled  down  in  order  to 
reduce  the  frustration  some  participants 
experienced  with  these  modules. 
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The  number  of  colleague  sign-offs  was  reduced 
and  the  nature  of  such  interactions  was 
restructured  to  make  them  more  helpful  to 
participants. 

Where  possible,  the  number  of  combat 
support  and  combat  service  support  tasks  was 
increased  in  the  module  examples  and  activi- 
ties to  enhance  the  perceived  applicability 
of  the  principles  to  a variety  of  MOS. 


CHAPTER  4 


EVALUATION  OF  WORKSHOP 


Evaluation  of  the  workshop  concerned  three  issues: 

. Are  the  materials  consonant  with  Guidelines 
for_  Deve lopment  of  SQT? 

. Can  TDA  personnel  complete  the  workshop 
within  the  time  limits? 

. IX)  TDA  personnel  perceive  the  workshop  as 
beneficial  to  their  assignment? 

Tlie  first  issue  was  settled  affirmatively  during  the  extensive 
reviews  of  the  initial  materials.  The  remaining  issues  were 
considered  throughout  the  implementation  phase. 


TIME  REQUIREMENTS 

The  14  workshops  (including  Leesburg)  were  attended  by  258 
people.  Twenty-five  of  the  participants  were  dropped  for  failing 
to  maintain  attendance.  Of  the  233  participants  who  completed  all 
requirements,  230  (98.7%)  finished  within  ten  working  days.  Days 
to  completion  ranged  from  three  to  13  working  days,  with  an  average 
completion  time  of  7.76  days.  These  results  indicate  that  the 
workshop  is  within  the  time  limit  that  had  been  established. 

PARTICIPANTS  1 REACTIONS 

As  participants  finished  the  workshop,  they  were  asked  to  com- 
plete an  end-o f-course  critique  form.  The  form  consisted  of  18 
statements  covering  various  aspects  of  the  workshop.  Participants 
indicated  their  reactions  by  responding  to  each  statement  on  a seven- 
point  scale,  from  "Strongly  Agree"  to  "Strongly  Disagree."  The 
critique  forms  were  completed  by  221  of  the  233  participants  who 
completed  the  14  iterations  of  the  workshop.  A copy  of  the  critique 
form  with  consolidated  reactions  is  presented  as  Appendix  A. 

The  first  item,  dealing  with  participants'  reactions  to  the 
introductory  briefing  delivered  by  project  staff,  elicited  varying 
responses.  Participants  who  had  not  been  in  any  self-paced  work- 
shop before,  particularly  in  the  early  workshops,  indicated  that 
thev  did  not  know  where  to  begin  after  the  briefing.  The  briefing 
was  revised  after  each  workshop,  until  in  the  last  iterations, 
participants'  responses  were  more  positive. 
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The  remaining  17  items  addressed  three  aspects  of  the  work- 
shop: format,  content,  and  impact.  A summary  of  participants’ 
responses  is  presented  in  Table  3,  with  the  17  items  grouped  by 
aspect.  It  is  apparent  from  the  table  that  the  responses  were 
highly  skewed  in  favor  of  the  workshop. 

The  mean  response  to  each  item  and  the  95%  confidence  interval 
about  each  mean  are  shown  in  Figure  1.  The  confidence  intervals 
indicate  the  range  of  values  within  which  we  are  fairly  certain 
the  true  mean  response  lies,  exclusive  of  sampling  error.  When 
the  interval  for  an  item  does  not  overlap  with  the  interval  for 
another  item,  we  may  conclude  that  the  means  for  those  items  are 
significantly  different.  Hie  width  of  the  interval  indicates  the 
distribution  of  responses  to  the  item:  the  wider  the  interval,  the 
greater  the  variability  among  responses.  Because  the  data  are  so 
highly  skewed,  it  is  not  possible  to  determine  what  aspects,  if 
any,  of  the  workshop  participants  did  not  like.  Therefore,  dis- 
cussion of  participants'  reactions  will  be  phrased  in  terms  of 
which  aspects  they  liked  more  or  less  than  others. 

Format.  Six  of  the  statements  on  the  critique  form  concerned 
the  format  of  the  workshop  and  mode  of  presentation.  Participants 
were  fairly  favorable  toward  the  sequencing  of  the  materials,  and 
toward  the  self-pacing.  There  were  more  negative  reactions  to  the 
self-pacing  than  to  any  other  single  aspect  of  the  workshop;  several 
participants  commented  that  pressures  from  their  offices  made  self- 
pacing impossible  for  them.  Reaction  to  the  style  of  presentation 
in  the  modules  was  mixed,  with  comments  indicating  that  some  mod- 
ules required  too  much  busy  work. 

Although  course  managers  did  point  out  that  colleague  inter- 
action was  permitted  at  any  time  during  the  workshop,  there  were 
few  specific  instructions  and  no  requirements  regarding  group  work. 
This  unstructured  approach  resulted  in  little  colleague  interaction, 
and  participants  commented  that  colleague  interaction  was  not  very 
helpful  to  them.  Course  managers  also  observed  that  participants 
were  reluctant  to  disturb  other  participants  with  questions  or 
ideas,  choosing  rather  to  talk  to  the  course  managers.  Participant 
reactions  to  the  availability  and  helpfulness  of  course  managers 
was  very  positive. 

Content.  Participants  were,  on  the  average,  less  positive 
about  the  content  of  the  workshop  than  they  were  about  format  or 
impact.  Some  participants  felt  that  the  examples  and  activities, 
while  plentiful,  did  not  clarify  all  of  the  principles.  They  also 
questioned  the  technical  soundness  of  the  material,  adding  that 
their  understanding  of  the  policy  had  differed  from  the  application 


16 


Table  3 


RESPONSES  TO  END-OF- COURSE 
CRITIQUE  ITEMS,  BY  ASPECT 


Aspect:  Format 

Respo 

Positive 

nse  (Percent)3 

Neutral  Negative 

2.  Logical  sequence  of  modules 

83.2 

14.5 

2.3 

6.  Style  of  modules  comfortable 

75.0 

21.8 

3.2 

7.  Set  own  pace 

84.1 

10.4 

5.5 

12.  Sufficient  colleague  interaction 

70.9 

25.5 

3.6 

13.  Colleague  interaction  helpful 

66.5 

29.4 

4.1 

14.  Sufficient  course  manager  interaction 

91.4 

8.1 

0.5 

15.  Course  manager  interaction  helpful 

95.0 

5.0 

0.0 

Average 

80.9 

16.4 

2.7 

Aspect:  Content 

Positive 

Neutral 

Negative 

3.  Sufficient  examples,  activities 

4.  Technically  sound,  consistent  with 

68.8 

26.2 

5.0 

pol icy 

75.9 

20.5 

3.6 

5.  Criterion  tests  measure  objectives 

9.  Sufficient  practice  in  activities. 

81.9 

15.8 

2.3 

tests 

81.0 

17.6 

1.4 

Average 

76.9 

20.0 

3.1 

Aspect:  Impact 

Positive 

Neutral 

Negative 

8.  Understanding  of  principles 

84.1 

14.5 

1.4 

10.  Mastery  of  skills 

66.1 

31.2 

2.7 

11.  Sense  of  achievement 

73.5 

24.7 

1.8 

16.  Important  to  present  job 

81.9 

15.4 

2.7 

17.  Course  interesting 

82.3 

15.4 

2.3 

18.  Worth  time  and  effort 

87.7 

11.4 

0.9 

Average 

79.3 

18.8 

2.0 

Responses  of  1 or  2 are  considered  Positive, 

responses 

of  3,  4 or 

5 

are  considered  Neutral,  responses  of  6 or  7 are  considered  Negative. 


ASPECT  AND  ITEM 
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Means  and  95%  Confidence  Intervals 
for  Responses  to  End-Of-Course  Critique  Items 
(N  = 221) 
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in  the  modules  (some  participants  had  not  had  access  to  the  Guide- 
lines before  the  workshop).  Reactions  to  the  criterion  tests 
were  somewhat  more  positive:  they  were  perceived  as  a fair  mea- 
sure of  the  objectives,  and  with  the  activities  provided  suffic- 
ient practice  in  the  skills  they  addressed. 

Impact . Participants  offered  mixed  reactions  to  the  state- 
ments regarding  the  impact  of  the  workshop.  Although  they  felt 
they  had  a fair  understanding  of  the  concepts  and  principles 

presented,  they  were  not  as  positive  about  their  mastery  of  the  j 

skills.  Participants  commented  that  they  were  reserving  judgment  | 

of  their  mastery  until  they  had  tried  their  newly-acquired  skills  j 

on  the  job.  Their  expression  of  a sense  of  achievement  fell  some-  ' 

where  between  the  positive  opinions  generated  by  their  understanding 
and  their  opinions  concerning  mastery.  For  the  most  part,  partici- 
pants felt  that  the  information  presented  was  important  to  them  in 
their  jobs — this  despite  the  fact  that  about  20%  of  them  were  not 
directly  involved  in  SQT  development.  Most  participants  found  the 
course  interesting.  Overall,  they  indicated  that  the  course  was 
worth  the  time  and  effort.  Although  previous  work  with  SQT  develop- 
ment ranged  from  no  experience  to  over  three  years,  the  amount  of  t 

experience  was  not  predictive  of  participants'  perceptions  of  the  . 

importance  or  worth  of  the  workshop.  | 

i 

Two  additional  questions  asked  whether  participants  would 
recommend  the  workshop  to  SQT  developers  and  whether  it  should  be 
added  to  the  school's  staff  and  faculty  development  program. 

Almost  all  participants  (98%)  endorsed  the  workshop  for  SQT  devel- 
opers. A smaller  majority  (87%)  thought  the  workshop  should  be 
instituted  at  their  school. 


i 

' CONCLUSIONS 

1 

{ 

: The  end-of-course  critique  data  provide  evidence  that  the 

1 project  objectives — to  design  and  implement  a workshop,  based  on 

the  Guidelines,  which  is  exportable,  self-paced,  and  can  be  com- 
pleted in  ten  days — were  realized. 

(The  first  characteristic  is  exportability,  in  this  case  the 

extent  to  which  the  workshop  can  be  conducted  by  TDA  personnel. 

At  each  of  the  13  workshop  iterations  after  Leesburg,  the  primary 
course  managers  were  members  of  the  staff  and  faculty  development 
, branch  of  the  local  TDA.  Although  they  were  still  in  probationary 

status,  they  were  responsible  for  setting  up  the  workshop  and  for 
bearing  an  equal  load  with  TDI  and  project  staff  personnel  in 
reviewing  criterion  tests.  Participants  indicated  positive  reac- 
tions to  the  contributions  of  the  course  managers  at  all  workshops. 
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with  no  different  i.-ition  made  between  TDI  and  project  staff  course 
managers  and  local  course  manager  trainees.  These  results  indi- 
cate that  TDA  personnel  who  complete  the  workshop  can  conduct  the 
workshop  successful ly. 

The  second  characteristic  evaluated  was  the  extent  of  self- 
pacing. Most  participants  (84Z)  agreed  that  they  set  their  own 
pace.  The  range  in  number  of  days  to  complete  the  workshop 
supports  their  contention. 

The  third  characteristic  was  whether  the  workshop  could  be 
completed  within  ten  days  even  though  participants  6et  their  own 
pace.  As  discussed  earlier,  the  workshop  is  well  within  that 
limit. 

In  the  final  analysis,  the  criterion  of  the  project’s  success 
would  be  that  workshop-certified  SQT  developers  consistently 
prepare  acceptable  SQT  products.  However,  SQT  products  are  almost 
always  the  result  of  contributions  from  subject  matter  experts 
and  testing  experts  who  are  not  workshop-certified,  as  well  as 
input  from  other  SQT  developers.  Therefore,  even  though  it  is 
possible  to  evaluate  products,  it  is  not  correct  to  attribute  the 
acceptability  (or  nonacceptability)  of  any  product  to  one  person. 

Because  of  the  absence  of  ftn  appropriate  external  criterion, 
evaluation  of  this  project's  success  focuses  on  the  internal  cri- 
teria that  the  workshop  can  be  implemented  within  the  constraints 
Imposed  and  that  participants  demonstrate  competency  in  the  skills 
in  the  Guidelines  before  they  finish  the  workshop.  These  criteria 
were  achieved. 


CHAPTER  5 


OVERVIEW  OF  WORKSHOP 


As  the  workshop  materials — modules,  criterion  tests  and  evalu- 
ation sheets — are  the  principal  result  of  the  project,  a brief 
overview  is  given  here.  The  materials  themselves  are  distributed 
within  the  Army  by  TRADOC  and  are  also  on  file  at  ARI. 


The  nine  major  tasks  discussed  earlier  were  grouped  into 
seven  phases  of  skill  development.  These  seven  phases  are  neces- 
sary for  complete  development.  Although  emphasis  in  the  workshop 
is  on  modules,  rather  than  phases,  this  overview  will  proceed  by 
phase. 

The  first  phase,  for  all  participants,  is  the  analysis  and 
planning  phase.  At  the  beginning  of  the  workshop,  participants 
select  an  MOS  with  which  to  work,  one  with  which  they  are  familiar. 
They  begin  with  ten  tasks  from  one  skill  level  of  that  MOS.  In 
one  module,  participants  identify  sources  of  information  on  each 
task  that  are  objective  indicators  of  need  for  evaluation.  In 
another  module,  participants  group  the  ten  tasks  according  to  the 
extent  of  known  performance  deficiencies.  These  modules  lead  par- 
ticipants to  select  for  testing  those  tasks  which  promise  the 
greatest  payoff  in  testing.  From  the  ten  tasks,  a course  manager 
then  selects  five  tasks  with  which  the  participant  continues  to 
work.  Then,  in  the  criterion  test  for  the  task  analysis  module, 
participants  review  and,  if  necessary,  revise  existing  task  analy- 
sis data  for  those  five  tasks  to  make  them  suitable  for  test  con- 
struction. The  final  module  in  the  analysis  and  planning  phase 
covers  allocating  tasks  to  components.  In  the  criterion  test, 
participants  assign  each  of  their  five  tasks  to  the  HOC,  the  PCC, 
or  the  WC.  High  skill  physical  tasks  are  allocated  to  the  HOC  or 
the  PCC;  mental  tasks  and  low  skill  physical  tasks  are  allocated  to 
the  WC. 

After  participants  finish  the  analysis  and  planning  modules, 
they  branch  into  either  the  HOC  construction  phase  or  the  WC  con- 
struction phase.  During  the  construction  phases,  participants 
work  with  the  tasks  selected  earlier.  For  the  HOC  construction, 
there  are  modules  for  some  preliminary  decisions  called  for  in  the 
Guidelines.  Then  they  work  on  modules  which  require  that  they  con- 
struct two  complete  hands-on  scorable  units,  to  include  performance 
measures,  conditions,  examinee  instructions  and  scorer  instructions. 

The  WC  construction  phase  also  requires  participants  to  write 
scorable  units  for  tasks  they  selected.  They  practice  constructing 
two  kinds  of  written  test:  written  performance  tests,  which  require 
examinees  to  perform  part  or  all  of  a task,  and  performance-based 
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tests,  which  require  examinees  to  answer  questions  about  how  a 
task  is  performed.  For  each  item  and  scorable  unit,  participants 
also  set  scoring  standards. 

After  participants  finish  the  construction  phase  for  a com- 
ponent, they  move  to  the  validation  phase  for  that  component.  Here, 
the  activities  and  criterion  tests  are  standardized,  and  address 
the  analysis  of  data  and  revision  of  scorable  units  based  on  vali- 
dation results. 

The  HOC  validation  procedures  check  interrater  reliability, 
acceptability,  and  feasibility.  The  modules  cover  locating  faults 
based  on  a tryout  with  experts,  computing  scorer  agreement,  checking 
feasibility  of  a scorable  unit,  constructing  a station-load  table, 
and  revising  hands-on  scorable  units. 

The  WC  validation  procedure  checks  discriminant  validity  and 
acceptability.  Three  options  for  validation  are  available,  based 
primarily  on  the  number  and  types  of  soldiers  to  which  the  developer 
has  access.  The  validation  modules  cover  collecting  self-ratings, 
locating  faults  based  on  a tryout  with  experts,  validating  written 
scorable  units  against  hands-on  tests,  selecting  a validation  option 
and  analyzing  data  on  each  of  the  three  options,  and  revising  written 
scorable  units. 

The  sixth  phase,  dealing  with  the  PC C,  focuses  not  only  on  the 
procedures  for  constructing  the  PCC  but  also  on  procedures  for 
validating  and  monitoring  it.  Participants  again  work  with  one  of 
their  own  tasks.  They  describe  how  the  test  will  be  conducted,  how 
it  will  be  validated,  what  kinds  of  results  would  indicate  units 
for  follow-up  checks,  and  how  the  checks  will  be  conducted. 

In  the  final  phase,  after  participants  have  developed  a scorable 
unit  for  each  component,  they  prepare  an  SQT  Notice.  This  is  pri- 
marily a check  on  their  mastery  of  the  format  for  the  Notice.  It 
also  provides  an  opportunity  to  clarify  any  confusing  aspect  of  the 
preceding  materials. 
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APPENDIX  A 


TRADOC  SQT  DEVELOPMENT  WORKSHOP 

We  are  interested  in  knowing  how  you  feel  about  several  aspects  of 
the  course  that  you  are  now  completing.  Our  aim  is  to  develop  courses 
which  are  interesting,  stimulating,  and  viewed  as  valuable  by  you. 

Your  honest  opinions  are  needed  so  that  we  can  refine  our  procedures 
and  materials. 

Thank  you. 

WORKSHOP  LOCATION: DATE: 

ORGANIZATION/SCHOOL : 

How  long  have  you  been  working  with  SQT?  Months 


Below  are  a series  of  statements.  Rate  each  as  to  whether  you  agree 
or  disagree  with  it.  (Feel  free  to  write  comments  about  your  answer  in 
the  margins  or  on  the  back  of  the  page . ) 


1.  After  the  introductory  briefing,  I knew  what  I was  supposed  to  do. 


Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 

2 

3 4 

5 

6 7 

Entry 

Mean 

Frequency 

71 

60 

32  22 

10 

6 6 

14 

2.43 

% 

34.3 

29.0 

15-5  10.6 

4.8 

2.9  2.9 

2.  The  modules  were  logically  sequenced. 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 

2 

3 4 

5 

6 7 

Entry 

Moan 

Frequency 

118 

65 

23  6 

3 

4 1 

1 

1.76 

ft 

53.6 

29-5 

10.5  2.7 

1.4 

1.8  0.5 

3 . There 
clear. 

were  sufficient  examples  and  activities  to  make  the 

information 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 

2 

3 4 

5 

6 7 

Entry 

Mean 

Frequency 

83 

69 

36  11 

11 

4 7 

2.27 

% 

37-6 

31.2 

16.3  5.0 

5.0 

1.8  3.2 

4.  The  information 

seemed 

to  be  technically  sound  and  consistent  with 

polii 

Strongly 

Agree 

Neutral 

Strongly 
Di sagree 

No 

1 

2 

3 4 

5 

6 7 

Entry 

Mean 

88 

40.0 


79 

35.9 


24 

10.9 


18 

8.2 


3 

1.4 


5 

2.3 


3 

1.4 


2.07 


Frequency 

% 


The  criterion  tests  were  a fair  measure  of  the  objectives. 


Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 

o 

3 

1* 

5 

6 7 

Fin  try 

Mean 

Frequency 

106 

75 

22 

8 

5 

1*  1 

1.86 

% 

1*8.0 

33.9 

10.0 

3.6 

2.3 

1.8  0.3 

6.  The  style  of  presentation  in  the  modules  was  comfortable. 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 

3 

1* 

5 

6 7 

Entry 

Mean 

Frequency 

ol* 

71 

23 

20 

5 

3 1* 

1 

1.33 

% 

1*2.7 

32.3 

10.5 

9.1 

2.3 

1.1*  1.8 

r.  I did 
pace . 

not  feel  pressured  to 

complete 

the  course  quickly;  I 

set  my 

own 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 

3 

1* 

5 

6 7 

Entry 

Mean 

Frequency 

160 

2ri 

11 

8 

1* 

3 9 

1 

1.71 

% 

72.7 

11.1* 

3.0 

3.6 

1.8 

1.1*  l*.l 

8.  I feel  that  I have  a 
presented. 

good  understanding  of 

the  concepts  and  princ 

iples 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

l 

O 

3 

U 

3 

6 7 

Entry 

Mean 

Frequency 

89 

96 

22 

10 

0 

2 1 

1 

1 .83 

e * 

1*0.  *5 

1*3.6 

10.0 

l*.5 

0.0 

0.9  0.3 

°.  The  activities  anil  criterion  tests  gave  me  sufficient  practice  in 
sharpening  my  skills. 

Strongly  Strongly 

Agree  Neutral  Disagree  N 


1 

2 3 

k 

5 

6 

7 

Entry  Mean 

Frequency 

91 

88  25 

12 

2 

2 

1- 

1.00 

% 

1*1.2 

39.8  11.3 

5.1* 

0.9 

0.9 

0.5 

10.  I feel  that  I have  a good  mastery  of  the  skills  offered. 


Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 2 

3 4 

5 

6 7 

Entry 

Mean 

Frequency 

-3- 

o 

H 

C\J 

1*5  21 

3 

5 1 

2.36 

% 

19.0  47.1 

20.1*  9.5 

1.4 

2.3  0.5 

11.  I feel  a strong  sense  of  achievement  as  a result  of  demonstrating 
my  competency  in  the  skills  offered. 

Strongly  Strongly 


Agree 

Neutral 

Disagree 

No 

1 

2 

3 

4 

5 

6 7 

Entry 

Moan 

Frequency 

71 

90 

30 

21 

3 

2 2 

2 

2.13 

% 

32.4 

4i.i 

13.7 

9.6 

1.4 

0.9  0.9 

12.  There 

was  sufficient 

colleague  interaction. 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 

2 

3 

4 

5 

6 7 

Entry 

Mean 

Frequency 

99 

57 

22 

27 

7 

7 1 

1 

2.14 

% 

45.0 

25-9 

10.0 

12.3 

3.2 

3.2  0.5 

13.  The  colleague  interaction  was  helpful  to  me 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 

2 

3 

4 

5 

6 7 

Entry 

Mean 

Frequency 

94 

53 

23 

31 

11 

6 3 

2.29 

% 

42.5 

24.0 

10.5 

14.0 

5.0 

2.7  1.4 

l4.  There 

was  sufficient 

Course  Manager  interaction. 

Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 

2 

3 

4 

5 

6 7 

Entry 

Mean 

Frequency 

157 

45 

12 

4 

2 

0 1 

1.43 

% 

71.0 

20.4 

5.4 

1.8 

0.9 

0.0  0.5 

15.  Interaction  with  the  Course  Manager  was  helpful  to  me. 


t 

\ 

I 


i 


.] 

t 


Strongly 

Agree 

Neutral 

Strongly 

Disagree 

No 

1 

2 

3 4 

5 

6 7 
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Frequency 

l6l 

1*9 
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1 
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22.2 

2.3  2.3 

0.5 

0.0  0.0 

16. 

The  information  presented  in  this  course  appears  to  be  very  important 
to  me  in  my  present  Job. 
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17. 

I found  the  course  interesting. 
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0.9 
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18. 

Considering  what  I knew  about  SQT  at  the  start  of  the  workshop  and 
what  I know  now,  the  workshop  was  worth  the  time  and  effort. 
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Would  you 

recommend 

this  course  to  other  SQT  developers? 

Yes 

214 

No 

(98.2*)  4 

(1.8*) 

No  Entry 

3 

Would  you  recommend  this  course  be  available  at  your  school  as 
the  internal  Staff  and  Faculty  development  program? 

part  of 

Yes  No 

186  (87.3*)  27  (12.7*) 

No  Entry 
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